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ABSTRACT 

Multi-sensor  tracking  potentially  has  many  advantages  over  single  sensor 
tracking.  This  report  evaluates  the  performance  of  a  multi-sensor  tracking 
algorithm,  the  asynchronous  fused  Kalman  filter,  using  both  simulated  and  real 
data  from  two  dissimilar  sensors.  The  real  data  was  collected  using  a  sensor- 
fusion  test-bed  consisting  of  two  sensors,  a  pulse  Doppler  radar  and  an  optical 
video  tracker.  The  performance  of  the  algorithm  has  been  evaluated  under 
various  conditions  including  clear  sky,  clutter,  multiple  targets  and  intermittent 
sensor  operation.  The  effect  of  sensor  fusion  on  the  system’s  robustness  to 
model  mismatch  has  also  been  investigated. 
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ASYNCHRONOUS  SINGLE  PLATFORM  SENSOR 

FUSION 


EXECUTIVE  SUMMARY 

Recent  trends  indicate  that  multi-sensor  fusion  will  become  an  essential  component  of 
military  platforms,  particularly  in  the  presence  of  electronic  countermeasures.  A  sound 
knowledge  and  understanding  of  sensor  fusion  techniques  is  essential  for  evaluating  future 
developments  and  acquisitions. 

A  multi-sensor  tracking  filter,  the  asynchronous  fused  Kalman  filter  (AFKF),  has  been 
developed  to  analyse  the  operation  and  performance  of  asynchronous  multi-sensor  trackers. 
This  algorithm  uses  a  variable  update  rate  Kalman  filter,  with  a  common  target  process 
model  to  represent  the  target  dynamics,  and  an  augmented  measurement  model  to  take 
account  of  multiple  sensor  inputs. 

A  sensor  fusion  test-bed  has  been  developed  using  the  generic  pulse  Doppler  radar 
(GPDR)  developed  by  the  Microwave  Radar  Division  of  the  Defence  Science  and  Tech¬ 
nology  Organisation,  Australia.  An  optical  tracking  system  has  been  added  to  the  radar, 
and  a  separate  computing  platform  installed  to  fuse  the  radar  and  optical  measurements. 

The  AFKF  algorithm  has  been  evaluated  using  both  simulated  and  real  data.  The 
simulated  data  demonstrated  the  domination  of  the  optical  measurements  on  the  angle 
trackers.  Under  ideal  conditions,  almost  identical  tracking  errors  are  achieved  by  a  Kalman 
filter  using  angle  measurements  from  the  optical  sensor,  and  range  and  Doppler  measure¬ 
ments  from  the  radar. 

Radar  and  optical  data  collected  using  the  sensor  fusion  test-bed  was  used  for  further 
evaluation  of  the  tracking  algorithm.  Discrepancies  between  the  filter  models  and  the 
actual  data  prevented  the  AFKF  from  achieving  the  results  obtained  with  the  simulated 
data.  The  data  revealed  that  the  optical  measurement  noise  increased  at  close  target 
ranges.  The  subsequent  model  mismatch  could  be  overcome  by  using  multiple  model 
techniques. 

The  AFKF  performed  well  with  intermittent  measurements,  easily  switching  between 
single  sensor  and  dual  sensor  operation.  However  its  performance  in  clutter  and  in  the 
presence  of  other  targets  was  poor.  The  optical  tracker  was  easily  seduced  from  the 
target  of  interest  by  other  targets  and  background  objects.  The  domination  of  the  optical 
measurements  on  the  operation  of  the  AFKF  caused  the  system  to  follow  the  seducing 
target,  subsequently  losing  track.  Some  form  of  data  association  is  required  to  overcome 
this  problem. 

The  effect  of  sensor  fusion  on  the  sensitivity  of  a  tracking  filter  to  model  errors  has 
been  investigated.  Generally  it  was  found  that  the  addition  of  another  sensor  only  affected 
the  sensitivity  when  the  noise  covariance  of  its  measurement  model  was  less  than  that  of 
the  original  sensor.  Under  these  conditions,  the  additional  sensor  dominated  the  tracking 
filter,  and  the  system  became  more  sensitive  to  model  errors. 
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1  Introduction 


Data  fusion  has  generated  significant  interest  in  recent  times.  Although  it  has  only 
recently  gained  popularity,  its  underlying  fundamental  technologies  are  well  established. 
It  is  more  accurately  described  as  a  new  application  of  existing  technologies,  rather  than 
a  new  technology.  One  particular  application  of  common  interest  to  both  civilian  and 
military  users  is  multi-sensor  tracking. 

A  multi-sensor  tracking  algorithm  has  been  developed  to  fuse  measurements  from  mul¬ 
tiple  collocated  dissimilar  sensors  (eg  radar,  and  infrared).  This  algorithm  was  designed 
to  operate  using  measurements  of  different  types,  eg  angle  only,  angle  and  range,  etc,  from 
unsynchronised  sensors.  The  measurements  arrive  from  different  sensors  at  different  times, 
at  different  rates,  and  possibly  intermittently. 

A  test-bed  has  been  developed  for  multi-sensor  data  collection  and  algorithm  testing. 
It  contains  two  sensors,  a  pulse  Doppler  radar  and  an  optical  video  tracker.  Further 
sensors,  such  as  an  ESM  receiver,  may  be  added. 

A  data  fusion  technology  base  has  been  established  through  collaboration  between 
various  divisions  of  the  Defence  Science  and  Technology  Organisation,  Australia  (DSTO), 
the  Cooperative  Research  Centre  for  Sensor  Signal  and  Information  Processing  (CSSIP), 
and  various  Australian  universities.  This  technology  base  will  provide  the  Australian 
Defence  Force  (ADF)  with  local  support  for  the  effective  evaluation  of  future  developments 
and  acquisitions. 


2  Defence  Significance 

Many  ADF  platforms  already  contain  multiple  sensors  with  varying  degrees  of  sensor 
integration  and  fusion.  Others  are  being  upgraded  with  additional  sensors  and  processing 
capability.  Sensor  fusion  has  the  potential  to  enhance  the  performance  of  locating  and 
identifying  targets  in  real,  hostile  and  cluttered  environments. 

The  utilisation  of  sensors  is  heavily  mission  dependent,  making  the  ADF’s  requirement 
unique.  It  may  therefore  be  necessary  to  develop  or  modify  sensor  systems  specifically 
for  the  Australian  environment  using  local  resources,  requiring  a  sound  knowledge  and 
understanding  of  data  fusion  techniques. 


3  Sensor  Fusion 


3.1  Overview 

Data  fusion  is  the  combination  of  information  from  multiple  sources  to  obtain  a  more 
accurate  and  complete  picture  of  the  environment  than  is  possible  from  a  single  source. 
Sensor  fusion  is  the  combination  of  data  from  multiple  sensors,  and  is  used  in  a  variety 
of  applications  including  tracking,  identification,  command  and  control,  manufacture  and 
assembly,  surveillance  and  robotics.  This  variety  of  applications  leads  to  the  use  of  a  range 
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of  technologies,  including  filtering,  artificial  intelligence,  knowledge-based  systems,  neural 
networks,  estimation  theory  and  wavelet  transforms. 

Fusing  sensors  of  similar  types  improves  accuracy  and  reliability  through  redundancy. 
Fusing  sensors  of  different  types  provides  additional  information,  giving  a  more  complete 
picture  of  the  environment.  For  example,  when  fusing  radar  and  optical  data,  the  higher 
angular  resolution  of  the  optical  sensor  is  combined  with  the  range  information  from 
the  radar.  Fusion  also  provides  a  mechanism  for  sensor  management  tasks,  such  as  sensor 
queuing  and  system  optimisation.  Combining  sensor  information  at  a  higher  level  presents 
an  overall  picture  and  assists  decision  making.  In  the  future  this  may  lead  to  automated 
decision  and  control. 

Data  fusion  can  be  viewed  as  a  hierarchical  structure,  as  illustrated  in  the  proposal 
by  the  Data  Fusion  Sub-panel  (DFS)  for  the  Joint  Directors  of  Laboratories  (JDL),  Tech¬ 
nical  Panel  for  C3  in  the  United  States  Department  of  Defense  (figure  1).  This  proposal 
introduces  three  successively  higher  levels  of  fusion,  with  sensor  management  providing 
feedback  for  sensor  control. 


Figure  1:  US-JDL  data  fusion  levels. 

Sensor  level  fusion  (level  1)  refers  to  directly  processing  the  measurements  or  informa¬ 
tion  from  the  sensors.  This  level  includes  tasks  such  as  detection,  parameter  estimation, 
tracking  and  identification. 

Situation  assessment  (level  2)  processes  the  outputs  from  level  1,  eg  target  tracks,  and 
determines  the  relationship  and  interaction  between  individual  objects.  These  relation¬ 
ships  and  interactions,  together  with  the  positional  information  from  level  1,  create  an 
overall  picture  or  model  of  the  environment,  in  particular  what  the  objects  are  doing  and 
how  they  are  doing  it. 

Threat  assessment  (level  3)  analyses  information  from  level  2,  determining  the  ex¬ 
pected  outcomes  and  possible  events  arising  from  the  situation.  It  considers  why  objects 
are  behaving  in  a  particular  way,  and  what  is  the  significance  of  this  behaviour.  This 
information  is  used  to  make  decisions  and  initiate  responses. 

Sensor  management  uses  the  outputs  from  the  above  levels  to  control  and  configure 
the  individual  sensors  from  a  system  point  of  view.  This  optimisation  is  dependent  on  the 
particular  application  and  operating  environment. 

This  report  addresses  the  level  1  function  of  target  tracking,  or  more  specifically,  the 
fusion  of  measurements  from  multiple  collocated  sensors  into  a  single  track.  This  task,  as 
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with  most  level  1  fusion  tasks,  may  be  separated  into  three  separate  sub-tasks,  namely 
positional  fusion  or  registration,  data  association  and  data  combination  (figure  2). 


Sensor  1 
Data 


Sensor  2 
Data 


Fused 

Data 


Figure  2:  Fusing  data  from  two  sensors. 

Positional  fusion  aligns  the  data  both  spatially  and  temporally.  This  guarantees  that 
all  sensors  are  looking  at  the  same  point  in  space  and  time.  This  task  is  generally  simpler 
for  single  platform  configurations,  because  the  relative  positions  of  all  sensors  are  known, 
and  reliable  temporal  synchronisation  of  the  sensors  is  easily  implemented. 

Data  association  determines  which  measurements  originate  from  a  target,  and  there¬ 
fore  may  be  used  for  tracking.  Nearest  neighbour,  Probabilistic  Data  Association  (PDAF), 
Fuzzy  Logic  and  Dempster-Schafer  are  examples  of  techniques  that  have  been  used  in  at¬ 
tempts  to  solve  this  problem.  This  is  perhaps  the  most  difficult  part  of  the  fusion  process, 
and  research  is  far  from  exhausted. 

Data  combination  is  the  combination  of  the  data  into  a  single  entity.  This  is  often  a 
weighted  sum,  with  the  weights  based  on  some  optimisation  criteria,  such  as  least  squares 
or  maximum  likelihood.  In  some  cases  the  combination  may  be  rule  based,  or  simply  a 
case  of  selecting  the  data  which  is  best  in  some  sense. 

Multi-sensor  tracking  provides  a  number  of  advantages  over  single  sensor  tracking. 
These  may  include: 

1.  improved  track  accuracy  through  availability  of  additional  and  possibly  more  accu¬ 
rate  measurements, 

2.  greater  reliability  through  sensor  redundancy, 

3.  increased  robustness  to  noise,  clutter  and  countermeasures,  and 

4.  stealth  operation  through  reduced  active  emissions  (ie  active  sensors  only  transmit 
when  absolutely  necessary). 


However  these  advantages  must  be  balanced  against  greater  system  complexity  and  cost. 


3 


DSTO-TN-0084 


4  Previous  Work 

The  majority  of  work  on  multi-sensor  tracking  has  centred  on  fusing  simultaneous 
measurements  from  two  or  more  sensors.  This  work  has  been  well  documented  in  the 
open  literature,  with  [1,  2,  3]  providing  useful  reading  on  the  subject.  Little  emphasis  has 
been  placed  on  the  specific  problem  of  asynchronous  fusion.  One  exception  is  the  work  of 
Blair,  Rice,  Alouani  and  Xia  [4,  5],  who  addressed  the  problem  of  asynchronous  fusion  for 
target  tracking.  They  extrapolated  the  measurements  to  a  common  time  before  combining 
the  data.  Extrapolating  reduces  the  computational  complexity,  but  also  reduces  the  track 
accuracy.  In  many  applications,  the  availability  of  increased  computing  power  has  removed 
the  need  for  extrapolation. 

The  problem  of  sensor  registration,  often  ignored  in  the  past,  has  been  receiving  in¬ 
creased  attention.  In  particular,  techniques  for  solving  this  problem  automatically  are 
being  pursued  at  the  CSSIP  [6]. 

The  use  of  a  Kalman  filter  with  augmented  measurements  and  a  common  dynamic 
model,  has  proven  a  useful  tool  for  combining  measurements  from  different  sensors.  How¬ 
ever,  the  most  difficult  problem  is  data  association.  This  problem  is  common  to  single 
sensor  multi-target  tracking,  and  has  attracted  considerable  interest  in  the  scientific  com¬ 
munity. 

Nearest  neighbour,  ie  using  the  measurement  located  closest  to  the  predicted  target 
position,  is  the  simplest  form  of  data  association.  Alouani,  Rice  and  Auger  [7]  developed 
a  generalised  nearest  neighbour  algorithm  that  uses  a  weighted  sum  of  all  measurements 
within  a  validation  gate.  These  weights  are  determined  by  the  distance  between  the 
measurement  and  the  predicted  target  position. 

Bar-Shalom  and  his  team  at  the  University  of  Connecticut  have  developed  the  Prob¬ 
abilistic  Data  Association  Filter  (PDAF)  [2].  This  filter  probabilistically  weights  all  mea¬ 
surements  within  a  validation  gate  according  to  their  distance  from  the  predicted  target 
position.  The  resulting  composite  measurement  is  used  to  correct  the  tracking  filter.  The 
equivalent  multi-target  tracker  is  known  as  the  Joint  PDAF,  but  is  computationally  very 
expensive.  Various  sub-optimal  alternatives  have  been  developed.  PDAF  methods  are 
particularly  suited  to  tracking  targets  in  clutter. 

Houles  and  Bar-Shalom  [8]  developed  a  multi-sensor  tracker  to  track  manoeuvring 
targets  with  a  radar  and  infrared  sensor.  This  algorithm  combines  the  PDAF  with  an 
interacting  multiple  model  (IMM)  tracker.  The  PDAF  handles  the  uncertainty  in  mea¬ 
surement  origin,  and  the  IMM  gives  the  algorithm  its  manoeuvre  handling  capability. 

Evans  and  others  [9]  at  the  University  of  Melbourne  have  developed  new  variations  on 
the  PDAF.  One  such  variation  is  the  Integrated  PDAF  [10],  which  treats  track  existence 
as  an  event  with  a  certain  probability. 

Deb  and  others  [11,  12]  partition  the  problem  into  a  set  of  feasible  measurement-to- 
target  associations.  They  choose  the  most  likely  partition  by  maximising  the  ratio  of  the 
likelihood  of  all  measurements,  given  the  partition,  to  the  likelihood  of  all  measurements 
being  false  alarms. 

Blackman’s  extensive  experience  in  association  and  fusion  of  multiple  sensor  data  in¬ 
cludes  multi-hypothesis  tracking  (MHT)  [13,  14].  This  algorithm  considers  every  possible 
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measurement  to  target  assignment,  and  therefore  grows  exponentially.  To  overcome  this 
impracticality,  least  likely  hypotheses  are  removed.  Blackman  has  developed  a  variation, 
called  most  probable  hypotheses  tracking,  where  all  significant  hypotheses  are  considered. 

Avitzour  [15]  used  a  maximum  likelihood  technique  to  solve  the  data  association  prob¬ 
lem.  He  treats  the  problem  as  one  of  missing  data,  where  the  measurement  to  target 
associations  are  the  missing  data.  Streit  and  Luginbuhl  [16]  have  combined  probabilistic 
and  maximum  likelihood  techniques  in  their  Probabilistic  Multi-hypothesis  Tracking  Al¬ 
gorithm  (PMHT).  This  algorithm  operates  in  a  single  sensor  multi-target  environment, 
estimating  the  assignment  probabilities  and  the  target  states  simultaneously.  This  work 
has  been  extended  to  multi-sensor  tracking  [17,  18,  19]. 

Hong  [20,  21]  has  worked  on  adaptive  sensor  fusion.  He  assumes  unknown  noise  co- 
variances,  and  uses  the  tracking  filter  innovations  to  determine  the  optimal  gains. 

Xie  and  Evans  [22]  have  used  hidden  Markov  model  tracking  for  multiple  targets.  They 
use  the  concept  of  a  mixed  track.  The  individual  tracks  are  easily  identified  in  the  mixed 
track.  Martinerie  and  Forster  [23,  24]  have  developed  a  two  stage  fusion  algorithm.  The 
first  stage  determines  a  distribution  of  likelihoods  over  a  discrete  space.  The  second  stage 
uses  hidden  Markov  model  techniques  to  discern  the  most  likely  track. 


5  Algorithm  Development 

5.1  Requirement 

In  a  real  sensor  suite  containing  multiple  sensors  of  different  types  (eg  radar  and  in¬ 
frared),  it  is  unlikely  that  the  measurements  from  all  sensors  will  arrive  at  exactly  the 
same  instant  in  time.  Different  sensors  are  likely  to  have  different  and  unsynchronised 
measurement  update  (sample)  rates.  Non-uniform  rates  may  arise  from  missed  measure¬ 
ments,  caused  by  either  faint  targets  or  deliberate  emission  control.  Dissimilar  sensors  may 
also  produce  different  types  of  measurements,  eg  bearing  and  range  from  a  radar,  and  just 
bearing  from  an  infrared  device.  I  will  refer  to  sensor  fusion  under  these  conditions  as 
asynchronous  sensor  fusion ,  or  more  simply,  as  asynchronous  fusion. 

The  sensors  are  assumed  to  be  collocated  on  the  same  mount.  As  the  spatial  relation¬ 
ship  between  the  sensors  is  fixed  and  known,  the  sensor  registration  problem  (positional 
fusion)  is  trivial  and  will  not  be  addressed  in  this  report. 


5.2  Tracking  Algorithm 

5.2.1  Kalman  Filter 

One  of  the  most  popular  tracking  filters  is  the  Kalman  filter  [1],  It  uses  two  linear- 
Gaussian  models,  a  process  or  dynamic  model  to  represent  the  target  dynamics,  and  a 
measurement  model  to  represent  the  measurements  received  from  the  sensor.  The  dynamic 
model  describes  how  the  target  state  changes  between  measurements,  and  is  expressed  as 

xtl+l  =  Ftlxtl  +  wti  (1) 
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where  at  time  U,  xti  is  the  target  state,  Fti  is  the  transition  matrix  describing  the  target 
dynamics  from  time  ti  to  tj+i,  and  u>t{  is  a  zero  mean  uncorrelated  Gaussian  process  noise 
with  covariance  Qti .  The  process  noise  models  deviations  of  the  target  from  the  expected 
dynamics,  eg  manoeuvres.  U  denotes  a  particular  time  instant,  where  t{  >  tj  for  i  >  j. 

At  each  filter  update  time,  ti,  the  measurement  model  is  defined  as 

zti  =  Htixti  +vti  (2) 

where  zt,  is  the  measurement,  Ht,  is  the  measurement  matrix,  and  u*.  is  a  zero  mean 
uncorrelated  Gaussian  measurement  noise  with  covariance  Rtr  The  measurement  matrix 
represents  the  relationship  between  the  measurement  and  the  target  state.  The  measure¬ 
ment  noise  includes  environmental  and  sensor  noise  in  addition  to  any  mismatch  between 
the  measurement  model  and  the  sensor. 

The  filter  produces  an  estimate  of  the  target  state  at  each  measurement  or  sample 
time.  The  state  estimate  obtained  at  time  ti  using  all  the  measurements  up  to  time  tj  is 
denoted  as  xti\ q,  and  is  obtained  using  a  two  step  process.  The  first  step  is  the  prediction 
step,  where  the  previous  state  estimate  and  the  dynamic  model  are  used  to  predict  the  new 
state  of  the  target,  |t. .  This  estimate  and  its  error  covariance,  Pq+1|q,  are  determined 
using  the  following  equations. 


PU+l\t  =  FtPt\tFZ  +  Qu  (3) 


The  second  step  is  the  correction  step,  where  the  measurement  received  at  time  t^x  is 
used  to  correct  the  predicted  state  estimate.  This  new  estimate,  xti+l\ti+l,  and  covariance, 
P(.+i|t  ,  are  determined  as  follows. 


=  pU+i\UHZ+i  +RU+1) 

=  ^ti+i\ti  +  Fti+l  (zti+ 1  —  Pti+i^ti+i\ti) 

Fti+\\tl+\  —  (l  —  Fti+iHti+1)Pti+1\tl 


(4) 


In  many  applications  measurements  arrive  at  equally  spaced  time  intervals,  and  the 
Kalman  filter  is  implemented  with  a  fixed  update  rate.  When  its  update  interval,  or 
time  between  measurements,  is  not  constant,  it  is  known  as  a  variable  update  rate  Kalman 
filter,  and  in  this  form  is  particularly  suited  to  asynchronous  sensor  fusion.  An  in  depth 
discussion  of  the  derivation  of  the  Kalman  filter  for  both  fixed  and  variable  update  rates 
can  be  found  in  [25]. 


5.2.2  Asynchronous  Fused  Kalman  Filter 

The  Asynchronous  Fused  Kalman  Filter  (AFKF)  is  a  multi-sensor  variable  update 
rate  Kalman  filter  with  a  single  state  space  dynamic  target  model,  and  a  time  variant 
augmented  measurement  equation.  At  any  measurement  time,  the  augmented  equation 
consists  of  a  stack  containing  the  individual  measurement  equations  for  each  sensor  pro¬ 
viding  measurements  at  that  time. 
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The  radar  and  optical  systems  measure  target  position  in  azimuth,  elevation  and  range, 
ie  radar  coordinates.  The  resulting  dynamic  model  is  non-linear,  with  complex  depen¬ 
dencies  between  these  positional  coordinates  and  their  derivatives.  For  non-manoeuvring 
targets  or  targets  at  long  range,  the  non-linearities  and  inter-relationships  can  be  modelled 
as  additional  process  noise.  Therefore  the  dynamic  model  becomes  linear,  with  a  state 
vector  denoted  (77,  77,  77,  e,  e,  e,  R,  R,  R,  )T,  where  77  represents  azimuth  or  bearing, 
s  represents  elevation,  and  R  represents  range,  (a,  and  a  indicate  the  first  and  second 
derivative  of  a  respectively.) 

The  AFKF  uses  measurements  from  either  the  radar,  optical  sensor  or,  if  the  mea¬ 
surements  happen  to  arrive  simultaneously,  both  together.  It  does  this  by  allowing  the 
measurement  model  (2)  to  vary  over  time.  If  the  measurements  arrive  from  the  radar, 
the  values  of  Htl  and  Rtt  represent  the  measurement  function  and  noise  covariance  of  the 
radar,  and  likewise  if  the  measurements  originate  from  the  optical  sensor. 

If  measurements  arrive  from  both  sensors  simultaneously,  the  measurements  and  their 
models  can  be  stacked,  ie 


ZU  = 


(r)' 

(o) 


lzU 


xti  + 


t,(r)' 

,>) 

JU  J 


Rt,  = 


R 


(r) 


<1 


where  the  superscripts  (r)  and  (o)  denote  the  appropriate  radar  and  optical  variables 
respectively.  If  simultaneous  measurements  are  rare,  as  is  the  case  for  the  radar  and 
optical  tracker,  it  is  simpler  to  process  them  using  a  zero  update  interval  in  the  Kalman 
filter. 


5.3  Adaptive  Tracking 

The  Kalman  filter  requires  the  covariance  of  the  measurement  noise  for  each  sensor, 
and  the  covariance  of  the  process  noise.  Generally  the  values  of  these  parameters  are 
assumed  to  be  known,  and  they  must  match  the  data  for  optimal  tracking.  The  sensor 
noise  covariance  depends  on  the  sensor  characteristics  and  the  environmental  conditions. 
The  process  noise  covariance  is  determined  by  the  mismatch  between  the  actual  target 
dynamics  and  the  assumed  model.  As  the  actual  covariances  all  vary  over  time  and  the 
selected  covariances  are  constant,  matched  conditions  cannot  be  maintained.  (The  effect 
of  model  mismatch  is  discussed  in  section  8.2.) 

An  alternative  is  to  estimate  these  parameters  from  the  data.  This  is  known  as  adaptive 
filtering,  and  a  number  of  techniques  have  been  investigated.  The  simplest,  explained 
by  Gelb  [26]  and  Mehra  [27],  estimates  the  actual  measurement  noise  and  state  error 
covariance  from  the  autocorrelation  functions  of  the  measurement  innovations  (the  errors 
between  the  measurements  and  the  predicted  target  state).  This  method  is  invoked  after 
filter  initialisation,  ie  when  the  Kalman  filter  has  reached  its  steady  state  condition.  Mehra 
outlines  this  technique  in  detail  for  a  constant  update  rate  Kalman  filter. 
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This  technique  becomes  significantly  more  complex  when  applied  to  asynchronous  sen¬ 
sor  fusion.  The  system  becomes  time  varying,  requiring  the  storage  of  previous  filter 
parameters  such  as  gains,  covariances,  etc,  and  the  autocorrelation  estimates  for  sensors 
with  low  measurement  rates  may  be  poor. 

In  practice,  not  all  elements  of  the  state  can  be  directly  measured.  This  coupled 
with  small  measurement  update  intervals  (order  of  tens  of  milliseconds)  may  cause  over 
estimation  of  the  process  noise  covariance.  This  introduces  additional  noise  into  the  output 
track  and  therefore  reduces  tracking  performance. 

Adaptive  systems  are  inherently  less  stable  than  fixed  systems,  increasing  the  likelihood 
of  the  filter  becoming  unstable  and  losing  track.  The  above  mentioned  over  estimation 
of  the  process  noise  covariance  is  a  likely  cause  of  filter  instability  in  applications  with 
high  measurement  rates.  Therefore  the  process  noise  covariance  is  allocated  an  assumed 
value  chosen  from  the  expected  target  dynamics.  During  steady  steady  state  operation, 
the  measurement  noise  covariance  is  usually  much  greater  than  the  state  estimate  covari¬ 
ance.  Therefore  the  measurement  noise  covariance  can  be  approximated  by  the  innovation 
covariance.  This  provides  reasonably  accurate  estimates,  and  hence  good  filter  stability 
over  a  wide  range  of  operating  conditions. 


6  Testbed 


6.1  Overview 

A  single  platform  sensor  fusion  test-bed  was  developed  to  collect  multi-sensor  data, 
and  to  test  sensor  fusion  algorithms  in  real  time  with  real  targets.  It  contains  two  sensors, 
a  pulse  Doppler  radar  and  an  optical  tracker,  both  physically  attached  to  the  same  mount. 
Additional  sensors,  such  as  infrared  and  ESM,  may  be  added  at  a  later  date. 

A  functional  diagram  of  the  testbed  is  shown  in  figure  3.  R,  t]  and  e  denote  the  range, 
azimuth  and  elevation  respectively,  and  errors  are  indicated  by  a  preceding  A.  A  dot 
above  the  parameter  symbol  indicates  a  rate  or  derivative,  and  estimated  parameters  are 
denoted  by  a  caret. 


6.2  Hardware 

The  generic  pulse  Doppler  radar  (GPDR)  is  an  experimental  pulse  Doppler  radar, 
operating  in  the  9-10  GHz  region  of  the  electro-magnetic  spectrum  (see  specification,  Ap¬ 
pendix  A).  It  was  developed  by  the  Microwave  Radar  Division  of  the  Defence  Science 
and  Technology  Organisation,  Australia  for  radar  evaluation  and  development.  The  radar 
is  housed  in  a  self  contained  trailer,  and  is  capable  of  operating  from  a  portable  3-phase 
generator  (figure  4) .  The  trailer  is  large  enough  to  house  the  optical  signal  processing 
equipment  and  the  fusion  processor. 

The  radar  is  almost  completely  software  controlled,  allowing  most  radar  parameters 
and  the  operator  interface  to  be  configured  by  software.  All  the  radar  parameters  can  be 
recorded  using  an  inbuilt  data  logging  facility. 
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A  colour  CCD  video  camera,  mounted  directly  above  the  radar  antenna  (see  specifica¬ 
tion,  Appendix  B),  is  used  as  the  optical  sensor.  It  is  housed  in  a  weather  proof  housing 
and  utilises  an  automatic  iris  control  to  prevent  damage  by  the  sun.  It  is  fitted  with  a 
75  mm  lens  through  a  focal  length  doubler,  giving  an  effective  focal  length  of  150  mm. 
This  gives  a  field  of  view  of  approximately  3.4°  in  azimuth  by  1.8°  in  elevation.  This  is 
comparable  with  the  radar’s  3dB  beam- with  of  approximately  1.6°.  The  two  sensors  are 
aligned  to  maximise  their  overlap. 

The  video  output  from  the  camera  is  fed  to  an  automatic  video  tracker  (AVT)  housed 
inside  the  GPDR  trailer.  The  ADEPT  20  Automatic  Video  Tracker  (see  specification, 
Appendix  C)  was  chosen  for  its  flexibility  and  level  of  control.  It  has  been  specially  mod¬ 
ified  to  provide  a  confidence  measure  for  its  position  outputs.  The  video  tracker  extracts 
the  target  from  the  video  image,  and  outputs  its  position  relative  to  boresight.  These 
measurements  are  available  on  a  frame  by  frame  basis.  Access  to  these  measurements, 
and  control  of  the  tracker,  is  achieved  through  either  a  serial  link  or  a  VME  interface. 

A  personal  computer  (PC)  is  used  to  control  the  AVT  and  display  its  output.  The  PC 
is  connected  to  the  AVT  through  a  PC/VME  interface  obtained  from  the  Bits  Corporation. 
The  VME  interface  provides  greater  access  to,  and  control  of,  the  AVT  than  is  possible 
through  the  serial  port. 

A  separate  PC  is  dedicated  as  the  fusion  centre.  In  addition  to  performing  the  fusion 
function,  the  fusion  centre  provides  an  operator  interface,  and  controls  the  GPDR  antenna 
mount  and  range  and  velocity  gates.  This  control  is  achieved  by  inserting  the  fusion  centre 
into  the  GPDR’s  control  loop.  A  16  bit  parallel  data  bus  provides  the  connection  between 
the  sensors  and  the  fusion  centre. 

Coarse  alignment  of  the  camera  to  the  radar  antenna  is  performed  as  part  of  the  radar 
calibration  procedure.  Fine  alignment  is  achieved  through  the  operator  interface. 


Figure  3:  Testbed  block  diagram. 
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Figure  4 •'  Sensor  fusion  testbed  on  location. 


6.3  Software 

A  software  library  has  been  developed  for  the  fusion  centre.  This  library  provides  an 
interactive  environment  that  allows  the  operator  to  control  the  system  and  view  its  status. 
In  particular  it  allows  the  operator  to: 

1.  create  a  new  or  load  a  previously  saved  system  configuration; 

2.  add  new  sensors  to  a  system  configuration; 

3.  modify  the  parameters  of  existing  sensors  (to  match  the  hardware); 

4.  connect  or  disconnect  individual  sensors; 

5.  initiate  and  terminate  tracking;  and 

6.  record  system  and  tracking  parameters. 

The  operator  display  consists  of  a  pull-down  menu  bar  across  the  top,  and  a  status  bar 
displaying  relevant  help  information  along  the  bottom.  The  remainder  of  the  screen  is 
split  horizontally,  the  upper  section  reserved  for  sensor  windows,  and  the  lower  containing 
a  parameter  display  window.  Each  sensor  in  the  system  is  identified  by  a  sensor  window  in 
the  upper  display  region.  This  window  contains  the  name  of  the  sensor,  its  configuration 
and  its  current  connection  status.  The  lower  parameter  display  window  contains  the  status 
of  various  system  and  track  parameters,  allowing  the  operator  to  observe  the  system  s 
operation.  The  operator  controls  the  system  using  keyboard  commands  and  pull-down 
menus. 


10 


DSTO-TN-0084 


The  library  also  performs  a  number  of  background  tasks  such  as: 

1.  transferring  measurements  from  the  sensors  to  internal  queues  ready  for  processing; 

2.  pulling  measurements  from  the  queues  in  time  sequence  for  processing; 

3.  transmitting  control  information  to  the  sensors;  and 

4.  extrapolating  the  tracking  filter  in  the  absence  of  measurements. 

The  library  does  not  include  the  fusion  algorithm.  This  is  the  responsibility  of  the 
system  developer,  which  is  incorporated  into  the  system  by  overloading  (or  overwriting) 
various  generic  functions  in  the  library.  Obviously  one  such  function  must  perform  the 
actual  fusion  or  tracking,  but  other  functions  are  available  for: 

1.  initialising  track  and  fusion  parameters; 

2.  terminating  tracking; 

3.  drawing  the  display  window; 

4.  placing  parameters  into  and  drawing  the  display  window; 

5.  sending  control  parameters  to  the  sensors; 

6.  selecting  parameters  to  be  recorded; 

7.  selecting  a  header  for  the  file  containing  the  recorded  data;  and 

8.  specifying  a  system  name  or  identifier. 

Other  functions,  such  as  those  for  pulling  sensor  measurements  from  the  queues  and  read¬ 
ing  the  status  of  system  parameters,  are  available  for  use  by  the  system  developer. 

The  GPDR  software  has  been  modified  to  include  a  software  interface  library.  The 
radar  has  been  configured  to  operate  in  either  stand  alone  or  fusion  mode,  the  mode  of 
operation  being  selected  by  the  operator.  When  used  in  fusion  mode,  the  radar  sends 
its  measurements  to  the  fusion  centre.  Its  servo  drives  and  range  and  velocity  gates  are 
controlled  by  data  received  from  the  fusion  centre  through  the  interface  bus. 

An  operator  interface  to  the  AVT  has  been  developed.  It  displays  the  current  tracker 
outputs  and  status,  and  provides  a  means  for  the  operator  to  control  and  configure  the 
AVT  through  a  series  of  menus  and  keyboard  commands.  The  operator  display  is  similar 
to  that  described  for  the  fusion  centre  but  without  the  sensor  windows.  The  outputs  from 
the  video  tracker  and  its  status  are  displayed  in  real  time.  The  operator  has  control  over 
a  number  of  AVT  functions  including: 

1.  automatic  or  manual  tracking  and  detection; 

2.  choice  of  tracking  algorithm; 

3.  choice  of  preprocessing  algorithm; 
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4.  tracking  window  position,  size  and  mode  of  operation; 

5.  conditions  that  must  be  met  before  a  detection  is  recognised  (eg  target  speed,  direc¬ 
tion,  size,  etc); 

6.  break-lock  and  coast  operation  (in  the  event  of  track  loss);  and 

7.  tracking  filter  parameters  for  filtered  outputs  if  used. 

This  interface  also  transmits  the  measurements  received  from  the  AVT  to  the  fusion  centre. 


7  Algorithm  Evaluation 

7.1  Overview 

The  AFKF  algorithm  has  been  evaluated  using  both  simulated  and  real  data.  The 
simulated  data  allowed  the  algorithm  to  be  tested  under  tightly  controlled  conditions, 
and  provided  ground  truth  or  true  target  position.  Each  track  consisted  of  measurements 
from  two  simulated  sensors.  The  real  data  was  collected  from  various  light  aircraft  and 
commercial  carriers,  and  contains  both  radar  and  optical  measurements  [28]. 

The  performance  of  the  AFKF  was  compared  to  the  performance  of  two  single  sensor 
Kalman  filters,  each  using  only  the  measurements  from  one  sensor.  Performance  com¬ 
parisons  were  based  on  the  covariance  of  the  track  errors,  obtained  from  the  target  state 
estimates  and  the  ground  truth. 


7.2  Simulated  Data 

The  results  from  the  simulations  verified  that  improved  tracking  performance,  ie  lower 
track  error  covariance,  is  obtained  by  fusing  measurements  from  an  additional  sensor.  The 
greatest  improvement  occurs  when  the  measurement  noise  covariance  of  both  sensors  are 
equal,  and  the  measurements  arrive  simultaneously  from  both  sensors.  When  the  sensor 
covariances  are  unequal,  the  tracking  performance  approaches  that  of  single  sensor  tracking 
using  the  sensor  with  the  least  noise.  In  the  case  of  the  radar  and  optical  sensor,  the  optical 
sensor  has  a  lower  noise  covariance  than  the  radar.  The  angular  tracking  performance  of 
this  system  is  therefore  only  marginally  better  than  that  of  the  optical  system  alone.  This 
suggests  that  only  the  range  and  Doppler  velocity  measurements  are  required  from  the 
radar. 

In  practice,  the  radar  has  a  lower  measurement  update  rate  than  the  optical  tracker. 
Simulations  under  this  condition  showed  a  decrease  in  performance  of  the  radar  only 
Kalman  filter.  The  AFKF’s  observed  performance  did  not  alter  because  of  the  much  lower 
noise  covariance  of  the  optical  sensor.  Theoretically,  the  AFKF’s  performance  would  have 
deteriorated  marginally. 

The  AFKF’s  performance  with  manoeuvring  targets  was  investigated.  Large  errors 
were  encountered  during  the  manoeuvres,  caused  by  significant  mismatch  between  the 
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data  and  the  algorithm’s  dynamic  model  during  the  manoeuvre.  Increasing  the  process 
noise  covariance  of  the  AFKF  reduced  these  errors,  but  also  decreased  the  straight  line 
tracking  performance.  Large  errors  may  result  in  track  loss  in  real  systems,  so  the  tracker 
design  is  a  compromise  between  straight  line  and  manoeuvring  targets. 

The  adaptive  algorithm  (section  5.3)  displayed  similar  results  to  those  obtained  from 
the  AFKF.  The  major  difference  was  an  increased  track  error  when  targets  manoeuvred, 
caused  by  an  overestimation  of  the  measurement  noise  covariance  during  the  manoeuvre. 


7.3  Real  Data 

7.3.1  Data  Collection 

The  data  used  in  this  study  was  obtained  by  locating  the  test-bed  at  an  elevated  site 
approximately  3km  from,  and  140  metres  above,  a  light  aircraft  airfield  situated  in  the 
northern  suburbs  of  Adelaide,  South  Australia.  This  site  also  provided  a  clear  view  of  the 
airspace  above  the  Adelaide  International  Airport,  some  25  km  away. 

The  collected  data  was  arranged  into  data  sets  containing  radar  and  optical  mea¬ 
surements  corresponding  to  single  target  tracks.  These  data  sets  include  aircraft  crossing, 
aircraft  landing  and  taking  off,  and  background  optical  and  radar  clutter  (buildings,  roads, 
etc).  The  video  tracker  was  not  available  during  data  collection,  so  the  optical  measure¬ 
ments  were  obtained  by  processing  the  recorded  imagery  through  the  video  tracker. 

The  true  target  position  is  required  to  determine  the  tracking  errors.  This  ground  truth 
was  not  available,  so  the  target  position  was  estimated  using  polynomial  fitting  techniques 
on  the  data.  This  smoothes  the  measurements  and  will  remove  any  short  time  constant 
variations,  such  as  those  produced  by  air  turbulence.  This  will  artificially  increase  the 
measured  track  error  covariance. 

During  data  collection,  the  absence  of  the  video  tracker  meant  that  the  sensor  mount 
was  controlled  solely  by  the  radar.  This  provided  the  opportunity  for  the  optical  system 
to  re-acquire  lost  tracks.  However  if  the  radar  lost  track,  the  optical  sensor  also  lost  track. 


7.3.2  Results 

The  AFKF  was  first  evaluated  under  near  ideal,  or  clear  sky,  conditions  in  which  clutter 
was  insignificant,  and  no  other  targets  were  present. 

The  AFKF  and  both  single  sensor  Kalman  filters,  ie  the  radar  Kalman  filter  and  the 
optical  Kalman  filter,  successfully  maintained  track  on  the  target  (aircraft)  of  interest. 
The  state  estimate  error  covariance,  ie  track  error  covariance ,  was  used  as  a  measure  of 
tracking  performance.  This  was  estimated  by  subtracting  the  estimated  ground  truth  from 
the  measured  covariance  of  the  state  estimate  error. 


Tables  1  and  2  show  typical  values  of  measured  track  error  covariance  obtained  from 
the  AFKF,  and  the  two  single  sensor  Kalman  filters,  for  both  light  and  commercial  aircraft. 
The  measured  results  do  not  achieve  the  expected  theoretical  values  [29],  probably  due  to 
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Table  1:  Track  error  covariance  of  a  light  aircraft. 


a  combination  of  mismatch  between  the  data  and  the  tracking  filter  models,  and  errors  in 
the  estimated  ground  truth. 

In  the  case  of  the  light  aircraft,  the  azimuth  track  errors  from  the  optical  Kalman 
filter  and  the  AFKF  are  consistent  with  the  dimensions  of  the  target.  The  greater  than 
expected  errors  result  from  the  optical  tracker  algorithm’s  inability  to  locate  a  consistent 
centroid  position  on  the  aircraft  body.  This  problem  is  particularly  evident  at  close  range, 
where  the  aircraft  occupies  a  significant  number  of  pixels  within  the  image.  This  addi¬ 
tional  measurement  noise  was  not  included  in  the  filter  models,  causing  a  severe  mismatch 
between  the  data  and  the  filter  models.  This  mismatch  degrades  the  performance  of  the 
optical  Kalman  filter  and  the  AFKF  to  a  point  where  their  performance  is  inferior  to  that 
of  the  radar  Kalman  filter  (Table  3).  This  is  caused  by  the  AFKF  virtually  ignoring  the 
radar  measurements,  and  relying  upon  the  now  much  noisier  optical  measurements. 


Table  3:  Track  error  covariance  of  light  aircraft  at  close  range. 


Tracker 

Azimuth  (mrad'*) 

Elevation  (rrirad2) 

Radar 

0.10 

0.021 

Optical 

0.34 

0.044 

Fused 

0.31 

0.040 

The  commercial  aircraft  provides  an  excellent  example  of  one  sensor  losing  the  target. 
In  this  case,  the  target  moves  out  of  the  optical  sensor’s  range  of  operation,  and  the 
AFKF  automatically  reverts  to  single  sensor  tracking  using  the  radar  measurements.  This 
is  illustrated  by  the  increase  in  azimuth  track  error  shown  in  figure  5,  where  the  optical 
measurements  ceased  after  125  seconds. 

The  elevated  site  provided  a  good  opportunity  to  evaluate  tracking  in  clutter,  as  the 
sensors  often  operated  against  an  urban  background.  The  optical  tracker  had  difficulty 
under  these  conditions,  frequently  acquiring  various  objects  in  the  background.  To  a  lesser 
extent,  the  radar  suffered  a  similar  fate  in  the  presence  of  large  reflectors,  eg  industrial 
sheds. 
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Figure  5:  Azimuth  tracking  error  increase  due  to  loss  of  optical  measurements. 

Figure  6  shows  the  azimuth  tracking  errors  obtained  from  a  target  which  was  tracked 
in  clutter.  The  dotted  line  represents  the  tracking  errors  obtained  from  the  optical  Kalman 
filter,  the  dashed  line  the  radar  Kalman  filter,  and  the  solid  line  the  AFKF. 


Figure  6:  Azimuth  tracking  errors  in  clutter. 

Initially,  all  trackers  were  following  the  target  in  a  relatively  clutter  free  environment. 
At  ‘A’  radar  clutter  was  introduced,  severely  degrading  the  performance  of  the  radar 
Kalman  filter.  The  AFKF  was  not  significantly  affected  because  of  the  dominance  of  the 
optical  measurements.  At  ‘B’  the  optical  sensor  loses  the  target,  and  the  AFKF  only 
receives  radar  clutter  measurements.  At  this  point  the  target  has  been  lost.  Optical 
clutter  seduces  the  AFKF  between  ‘C’  and  ‘D\  After  a  few  spurious  false  alarms  from 
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both  sensors,  radar  measurements  recommence  at  ‘E\  However  these  are  not  from  the 
target,  and  cease  at  ‘F’. 

A  second  example  is  illustrated  by  the  azimuth  and  elevation  tracks  in  figure  7.  In 
this  scenario,  radar  clutter  first  appears  at  ‘A’.  As  in  the  previous  example,  the  AFKF 
maintains  track  using  the  optical  measurements.  At  ‘B’  the  radar  is  seduced  by  clutter, 
believed  to  originate  from  a  large  iron  building  near  the  bottom  edge  of  the  main  beam. 
The  AFKF  continues  tracking  the  target  until  ‘C\  when  all  measurements  cease  because 
the  mount  has  moved  the  sensors  away  from  the  target.  At  ‘D’  a  few  spurious  radar 
measurements  influence  the  AFKF  and  radar  Kalman  filter  until,  at  ‘E’,  the  target  fortu¬ 
itously  re-enters  the  field  of  view,  and  is  re-acquired  by  the  optical  tracker.  The  AFKF 
and  optical  Kalman  filter  then  continue  to  track  the  target. 


(a)  Azimuth. 


(b)  Elevation. 


Figure  1:  Tracking  in  clutter. 

In  general,  it  was  more  common  for  the  optical  tracker  to  be  effected  by  clutter  and 
subsequently  cause  system  track  loss.  This  is  because  the  optical  sensor  is  two-dimensional, 
and  therefore  has  no  range  or  depth  information.  Therefore  it  receives  measurements 
from  both  the  target  and  clutter.  As  a  result,  the  AFKF  performed  poorly  in  clutter 
because  of  the  strong  influence  of  the  optical  measurements  on  its  operation.  When  optical 
measurements  were  available,  radar  clutter  had  little  effect  on  the  AFKF’s  performance 
because  the  radar  measurements  contributed  little  to  the  fused  track. 

Other  targets  passing  in  front  of  the  target  of  interest,  and  large  targets  in  general, 
often  seduced  the  optical  tracker  and  the  AFKF  from  the  target  of  interest.  An  example 
demonstrating  this  effect  is  illustrated  by  the  measurements  and  fused  track  (dashed  line) 
shown  in  figure  8.  The  track  commences  at  ‘A’  and  proceeds  in  a  clockwise  direction.  The 
AFKF  is  seduced  from  the  target  by  clutter  at  ‘B’,  and  other  targets  at  ‘C’  and  lD’.  These 
other  targets  are  lost  when  they  leave  the  optical  field  of  view.  After  losing  the  target  at 
‘D’,  the  optical  tracker  is  seduced  by  several  interferences  near  the  top  of  the  field  of  view 
(‘E’)  before  re-acquiring  the  target.  Normal  dual  sensor  tracking  continues  until  the  radar 
loses  the  target  (‘F’),  at  which  point  the  AFKF  tracks  the  target  using  only  the  optical 
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measurements  until  the  radar  re-acquires  the  target  (‘G’). 
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(a)  With  radar  measurements. 


(b)  With  optical  measurements. 


Figure  8:  Azimuth- elevation  track  with  interfering  targets. 

The  radar  only  (dotted  line),  and  fused  (solid  line)  plots  from  a  second  example  are 
shown  in  figure  9.  Commencing  from  the  right,  both  sensors  track  the  target  until  the 
optical  sensor  is  seduced  by  another  optical  target  at  ‘A’.  The  AFKF  track  follows  this 
target  until  it  is  lost  by  the  optical  sensor.  The  AFKF  track  attempts  to  re-align  itself 
with  the  radar  measurements,  but  is  prevented  by  several  short  term  optical  interferences 
at  ‘B’,  ‘C’  and  ‘D’.  Although  they  only  exist  for  a  short  period,  the  dominance  of  the 
optical  measurements  is  sufficient  for  them  to  have  a  significant  effect  on  the  AFKF  track. 
Therefore  once  a  track  is  seduced,  it  may  take  only  relatively  minor  interferences  to  prevent 
re-establishment  of  the  correct  track. 


Figure  9:  Azimuth-elevation  track  with  interfering  targets. 

In  real  time  operation,  track  loss  will  occur  once  the  optical  tracker  has  been  seduced. 
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This  will  occur  because  the  optical  measurements  force  the  AFKF  to  move  the  mount 
away  from  the  target. 

To  simulate  an  offset,  or  registration,  error  between  the  radar  and  optical  sensor,  a  5 
milli-radian  azimuth  offset  was  added  to  the  optical  measurements.  The  plot  in  figure  10 
shows  the  effect  on  the  track  azimuth,  where  the  dashed  line  represents  the  radar  Kalman 
filter  track,  the  dotted  the  optical  Kalman  filter  track,  and  the  solid  the  AFKF  track.  The 
AFKF  track  falls  between  the  two  sets  of  measurements,  its  actual  position  determined 
by  the  relationship  between  the  noise  covariance  of  the  two  sensors.  The  AFKF  track  will 
be  closer  to  the  sensor  with  the  lowest  noise  covariance;  therefore  if  the  optical  sensor  is 
correctly  aligned,  the  error  is  relatively  small.  However  if  the  radar  is  correctly  aligned, 
and  the  optical  sensor  is  misaligned,  the  error  is  much  larger.  Therefore  it  is  important  to 
align  the  higher  resolution  sensors  accurately. 


Figure  10:  Azimuth  tracking  error  with  Figure  11:  Track  error  caused  by  jet 
optical  misalignment.  vapour  trail. 

The  test-bed  attempted  to  track  a  Boeing  737  commercial  aircraft  producing  a  distinct 
vapour  trail.  The  radar  maintained  track  on  the  body  of  the  aircraft.  However  the  optical 
tracker  could  not  maintain  a  steady  centroid  position  on  the  target,  preferring  to  move 
randomly  along  the  aircraft  body  and  the  vapour  trail.  As  the  optical  measurements 
dominate  the  AFKF,  the  fused  track  also  moved  along  the  vapour  trail,  creating  large 
track  errors.  Figure  11  shows  the  error  between  the  AFKF  and  the  radar  Kalman  filter 
tracks.  The  fused  track  lags  an  average  of  30  metres  behind  the  radar,  reaching  distances 
of  up  to  80  metres  at  times.  In  particular,  the  last  half  of  the  track  shows  large  excursions 
along  the  vapour  trail. 

Two  aircraft  were  observed  flying  elliptical  circuits  in  formation.  While  tracking,  the 
radar  frequently  switched  between  targets.  This  is  particularly  likely  when  targets  are 
separated  in  azimuth  but  not  range,  because  the  radar’s  angular  discrimination  is  relatively 
poor.  The  higher  angular  resolution  of  the  optical  system  enabled  it  to  maintain  track  on 
a  single  aircraft,  but  it  often  changed  targets  as  they  crossed  in  the  field  of  view.  Although 
this  caused  all  three  trackers  to  produce  noisy  tracks,  they  all  maintained  track  on  the 
formation.  Therefore  although  the  optical  tracker  may  be  able  to  discriminate  between 
individual  targets  in  a  formation,  it  may  not  be  able  to  maintain  track  on  a  particular 
target  if  the  formation  is  manoeuvring. 
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The  adaptive  algorithm  (section  5.3)  was  also  evaluated  using  the  real  data.  In  general 
its  performance  was  similar  to  that  of  the  AFKF.  However  it  showed  a  lower  tendency  to 
be  seduced  by  interfering  targets.  Under  certain  conditions,  the  predicted  target  position 
does  not  immediately  follow  the  interfering  target.  This  causes  the  estimate  of  optical 
measurement  noise  covariance  to  increase  and  reduce  the  contribution  of  the  optical  sen¬ 
sor.  This  allows  the  AFKF  to  follow  the  radar  measurements  and  continue  tracking  the 
target.  However  the  number  of  lost  tracks  due  to  seduction  of  the  optical  sensor  was  still 
unacceptable. 


8  Model  Mismatch 


8.1  Overview 

Successful  tracking  requires  a  number  of  assumptions,  particularly  with  regard  to  the 
expected  dynamics  of  the  target  and  the  characteristics  of  the  measuring  sensors.  This 
knowledge  is  expressed  using  dynamic  and  measurement  models  respectively.  Of  particular 
interest  to  the  tracking  filter  designer  is  the  effect  of  mismatch  between  the  assumed  models 
and  the  actual  scenario.  This  is  often  referred  to  as  sensitivity  analysis.  Sensitivity  analysis 
has  been  covered  extensively  for  single  sensor  systems,  and  in  particular  for  the  state  space 
based  Kalman  filter  [26]. 

This  section  evaluates  the  effect  of  adding  an  additional  sensor  on  the  robustness  of 
the  tracking  filter  to  model  mismatch,  or  alternatively,  on  the  sensitivity  of  the  tracking 
filter  to  model  errors. 

8.2  Sources  of  Model  Mismatch 

Model  mismatch  may  occur  in  either  the  state  transition  matrix  or  the  measurement 
matrix  [26].  For  linear- Gaussian  filtering,  the  dynamic  model  mismatch  may  occur  in  the 
state  transition  matrix  ( F ).  However,  any  mismatch  here  is  generally  approximated  by 
additional  process  noise.  The  assumption  of  Gaussian  noise  preserves  the  linearity  of  the 
filter,  and  the  effect  of  non-Gaussian  noise  is  not  considered.  Also  the  effect  of  bias  on 
this  noise  is  ignored.  The  process  noise  covariance  is  a  critical  design  parameter  of  the 
Kalman  filter,  and  the  effect  of  errors  in  its  value  is  considered.  If  the  assumed  value  is 
too  high,  the  filtered  state  estimate  will  contain  excessive  noise.  A  value  which  is  too  low 
will  ultimately  cause  divergence  from  the  true  track. 

The  measurement  function  ( H )  is  another  potential  source  of  model  mismatch.  This 
model  may  be  a  linear  approximation  of  a  non-linear  coordinate  transformation.  Again 
this  may  be  approximated  by  additional  measurement  noise.  As  for  the  process  noise, 
the  effect  of  non-Gaussian  noise  is  not  considered,  and  it  is  assumed  that  any  bias  can  be 
effectively  removed  through  correct  sensor  registration.  The  measurement  noise  covariance 
is  also  a  critical  design  parameter.  An  excessive  design  value  will  reduce  the  Kalman  gain, 
increasing  the  possibility  of  track  divergence.  A  smaller  value  will  increase  the  gain, 
introducing  additional  sensor  noise  into  the  state  estimate. 
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8.3  Evaluation 

8.3.1  Performance  measures 

To  evaluate  the  effect  of  the  measurements  from  the  additional  sensor,  it  is  first  nec¬ 
essary  to  remove  the  effect  of  fusion  from  the  track  error  covariance.  Assuming  that  the 
average  or  mean  tracking  error  is  zero,  a  useful  measure  of  tracking  performance  is  the 
covariance  of  the  tracker’s  position  errors,  or  track  error  covariance.  1  The  lower  the  track 
error  covariance,  the  better  the  tracking  performance.  The  improvement  in  tracking  due 
to  sensor  fusion  can  then  be  defined  as  the  ratio  of  the  track  error  covariance  of  a  single 
sensor  tracker  to  the  track  error  covariance  of  the  dual  sensor  tracker,  referred  to  here 
as  the  fusion  gain  ( FG ).  If  the  FG  is  determined  when  both  the  single  and  dual  sensor 
trackers  are  operating  under  matched  conditions,  it  is  referred  to  as  the  matched  FG.  A 
FG  greater  than  unity  implies  an  improvement  in  tracking  performance,  and  a  value  below 
unity,  a  degradation  of  tracking  performance. 

The  mismatched  FG  provides  a  measure  of  the  change  in  tracking  performance  of  a 
mismatched  tracker  caused  by  the  addition  of  another  sensor.  Of  particular  interest  is 
the  comparison  between  this  and  the  corresponding  change  under  matched  conditions.  A 
mismatch  factor  ( MF )  is  introduced  as  the  ratio  of  mismatched  FG  to  matched  FG.  The 
MF  is  a  measure  of  the  improvement  obtained  in  the  tracking  performance  of  a  mismatched 
tracker  by  adding  a  second  sensor,  ignoring  the  expected  improvement  due  to  the  fusion 
process  alone.  This  is  effectively  a  measure  of  the  improvement  in  the  tracker’s  sensitivity 
to  model  mismatch,  with  values  greater  than  unity  indicating  reduced  sensitivity  to  model 
mismatch,  and  values  below  unity  indicating  increased  sensitivity. 


8.4  Results 

8.4.1  Simulations 

A  selection  of  single  sensor  and  dual  sensor  Kalman  filters  with  various  design  values  of 
process  noise  covariance  and  measurement  noise  covariance  were  applied  to  two  hundred 
sets  of  statistically  equivalent  data.  The  average  steady  state  track  error  covariance  was 
determined  for  each  filter. 

The  effect  of  process  model  mismatch  was  evaluated  by  varying  the  design  value  of  the 
filter  process  noise  covariance  while  matching  the  measurement  noise  covariance  to  the 
data.  Similarly,  the  effect  of  measurement  noise  mismatch  was  evaluated  by  varying  the 
filter  design  value  of  the  measurement  noises  while  maintaining  matched  process  noise. 


8.4.2  Process  Noise  Mismatch 

The  results  in  figure  12  show  that  the  MF  is  approximately  unity  for  design  process 
noise  covariances  over  the  range  from  0.001  to  1000  times  that  of  the  data.  This  indicates 

'Under  zero  average  track  error  conditions,  the  track  error  covariance  is  defined  as  the  average  squared 
track  error. 
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that  the  contribution  of  the  additional  sensor  to  the  sensitivity  of  the  system  to  process 
noise  covariance  mismatch  is  insignificant. 


Log  Process  Noise  Mismatch 


Figure  12:  Mismatch  factor  for  process  noise  mismatch 


8.4.3  Measurement  Noise  Mismatch 

This  section  considers  various  levels  of  model  mismatch  in  both  sensors.  The  sensor 
used  in  the  single  sensor  system  is  referred  to  as  the  original  sensor ,  and  the  sensor  added 
to  it  to  form  the  dual  sensor  system  is  referred  to  as  the  additional  sensor.  The  results  are 
displayed  as  three  dimensional  meshes  with  an  underlying  contour  plot  for  clarity.  The 
vertical  axis  indicates  the  value  of  either  the  logarithm  of  the  matched  FG,  or  the  MF . 
The  other  two  axes  indicate  the  level  of  model  mismatch  in  each  sensor,  where  the  scale 
represents  the  logarithm  of  the  ratio  of  the  filter  design  measurement  noise  covariance  to 
the  actual  measurement  noise  covariance  of  the  data,  eg  2  denotes  a  design  covariance  of 
100  times  that  of  the  data,  and  zero  denotes  the  matched  condition.  A  detailed  discussion 
of  these  results  may  be  found  in  [29]. 

The  mismatch  factor  for  equal  measurement  noise  covariance  in  both  sensors  is  shown 
in  figure  13.  These  results  show  that  fusing  the  measurements  from  an  additional  sensor 
whose  design  measurement  noise  covariance  is  greater  than  or  equal  to  the  original  sensor 
(ie  towards  the  right  of  the  plot),  does  not  alter  the  system’s  robustness  to  measurement 
model  mismatch.  However,  if  the  design  noise  covariance  of  the  additional  sensor  is  less 
than  that  of  the  original  (ie  towards  the  left  of  the  plot),  the  fused  system’s  performance 
is  poorer  than  that  expected  from  the  filter  design  equations.  Under  these  conditions  the 
system  becomes  more  sensitive  to  modelling  errors. 

Figure  14  illustrates  the  results  when  the  data  received  from  the  additional  sensor  has 
a  noise  covariance  ten  times  that  of  the  original.  It  can  be  seen  that  these  results  are 
similar  to  those  obtained  when  the  measurement  noise  covariances  are  equal. 


The  results  obtained  when  the  noise  covariance  of  the  data  from  the  additional  sensor 
is  ten  times  smaller  than  that  of  the  original  sensor  are  shown  in  figure  15.  The  MF  is 
again  similar  to  those  obtained  from  the  equal  measurement  noise  covariances,  with  the 
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original  sensor  mismatch  -3  -3  . 

°  additional  sensor  mismatch 


Figure  13:  Mismatch  factor  for  sensors  with  equal  noise  covariance. 


Figure  14:  Mismatch  factor  when  the  additional  sensor  is  noisier  than  the  original. 


exception  of  a  peak  of  approximately  1.7  when  the  design  measurement  covariances  are 
equal.  These  above  unity  values  of  MF  indicate  that,  under  these  conditions,  the  system’s 
sensitivity  to  model  errors  decreases. 


9  Conclusions 


An  asynchronous  fusion  algorithm,  the  asynchronous  fused  Kalman  filter  (AFKF),  has 
been  developed  using  a  variable  update  rate  Kalman  filter  with  an  augmented  measurement 
model.  This  algorithm  has  been  evaluated  using  both  simulated  and  real  data.  An  adaptive 
version  has  also  been  evaluated. 

A  sensor  fusion  test-bed  has  been  developed  for  multi-sensor  data  collection  and  al- 
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Figure  15:  Mismatch  Factor  when  the  original  sensor  is  noisier  than  the  additional. 


gorithm  evaluation.  It  currently  consists  of  two  sensors,  a  pulse  Doppler  radar  and  an 
optical  video  tracker. 

The  asynchronous  fused  Kalman  filter  (AFKF)  performed  well  under  clear  sky  condi¬ 
tions.  Its  performance  deteriorated  at  close  range  because  of  increased  optical  noise. 

The  AFKF  performed  well  with  intermittent  sensor  operation.  When  the  measure¬ 
ments  from  one  sensor  ceased,  the  AFKF  reverted  to  single  sensor  tracking,  using  the 
measurements  from  the  remaining  sensor. 

The  AFKF  performed  poorly  in  clutter  and  in  the  presence  of  interfering  targets,  often 
losing  track.  The  optical  sensor’s  susceptibility  to  clutter  and  other  interfering  targets  was 
caused  by  its  lack  of  range  discrimination.  To  overcome  this  problem,  some  form  of  data 
association  is  required  to  determine  which  measurements  should  be  used  for  tracking  (eg 
Multi-Sensor  Probabilistic  Multi-Hypothesis  Tracking  [17,  18]). 

When  tracking  in  clutter  or  at  close  range,  the  ability  to  change  the  tracking  filter 
models  to  suit  the  conditions,  ie  adaptive  or  multiple  model  techniques,  would  be  advan¬ 
tageous. 

The  AFKF  successfully  tracked  a  formation  of  two  aircraft.  However,  although  the 
optical  sensor  was  able  to  discriminate  between  the  two  aircraft,  it  frequently  switched 
targets  during  manoeuvres. 

The  addition  of  another  sensor  does  not  affect  the  sensitivity  of  the  tracking  filter  to 
errors  in  the  noise  covariance  in  the  process  or  dynamic  models.  In  general,  the  addition 
of  another  sensor  only  affects  the  sensitivity  of  the  tracking  filter  to  errors  in  measurement 
noise  covariance  when  the  tracking  filter  measurement  noise  covariance  for  the  additional 
sensor  is  less  than  that  for  the  original  sensor.  In  this  case,  the  system  becomes  more 
sensitive  to  errors  in  the  measurement  models. 
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Appendix  A 

Generic  Pulse  Doppler  Radar  Specification 


Antenna 

Prime  focus  fed  amplitude  monopulse 

Diameter 

Gain 

3  dB  sum  beamwidth 

Antenna  Mount 

Analogue  rate  loops  in  Azimuth  and  Elevation 
Small  signal  rate  rise  time 

Angle  resolution 

Transmitter 

Peak  power 
Max  duty  cycle 

Receivers 

Three  matched  channels 
Operating  frequency 
Noise  figure 

Analogue  to  digital  converter  resolution 
Max  sample  rate  (I  and  Q  data) 

AGC  range 
AGC  resolution 

Timing  and  Control 

Pulse  repetition  interval  (PRI) 

Pulse  length 

Block  length  (coherent  interval) 

Range  gate  resolution 

Tracking  parameters 

Target  range  cells 
Noise  range  cells 
Doppler  filters  per  range  cell 
Pulse  repetition  frequency  (PRF) 

Filter  cycle  time 


1500mm 
39  dB 
1.6° 


10  ms  (az) 

15  ms  (el) 

0.383 mrad  (Mbit) 


200  W 
30% 


9.0  -  10.0  GHz 
<5dB 
12  bits 

8xl06  samples/s 

78  dB 

2dB 


5  -  255  /is 
1  /is 

1.4  -  65  ms 
62.5  ns 


2  per  channel 
4  in  sum  channel 
128 

13.157  -  26.315  kHz 
33.3  ms 
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Appendix  B 

Video  Camera  Specification 


Configuration 

Colour  CCD  video  camera  with  75  mm  automatic  iris  lens  and  2x  extender 
Field  of  view  (FOV)  2.48°  x  1.87° 


Camera 

Pick-up  device 
Picture  element 
Scanning  area 
Scanning  system 
Scanning  frequency 

Resolution 

Signal  to  noise  ratio  (SNR) 
Output 


\  colour,  interline-transfer  CCD 
681  (H)  x  582  (V) 

6.5  mm  (H)  x  4.9  mm  (V) 

2:1  interlaced 
15.625  kHz  (H) 

50  Hz  (V) 

>  430  TV  lines  (H) 

>  420  TV  lines  (V) 

>  48  dB 

VBS  l.OVp-p  PAL  compatible 
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Appendix  C 

Adept20  Automatic  Video  Tracker 


Video  Input 

Format 

Level 

Standards 

Type 

Impedance 

Video  Output 

Normal  video  with  symbology  overlay 

Format 

Level 

Impedance 

Video  Preprocessing 

Spatial  enhancement 

Statistical  enhancement 

Threshold  (positive,  negative  contrast) 

Track  Algorithms 

Centroid 

Correlation 

Track  Window  Position 
Manual 

Automatic  (follows  target) 

Track  Window  Size 

Manual 

Adaptive  (automatically  encloses  target) 

Breaklock/ Coast 

Automatic  breaklock  algorithm 

Automatic  target  re-acquisition  after  breaklock 


Specification 


Composite  Video 
635/525  line 
1.0  Vp-p 
CCIR  or  RS170 
Differential 
75  n 


Composite  Video 
1.0  Vp-p  (into  75  ft) 
75  n 
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Performance 

Typical  for  target  size  >6x6  TV  lines 

Min  target  contrast 

Min  signal  to  noise  ratio 

Noise  on  target  position  output 

Error  update  rate 

Error  latency 

VMEbus  Interface 


<  5% 

<  4 

<  1  TV  line  (3  a) 
50  Hz 

<  15  ms 


Capability 
Board  select 


Slave,  A24,  D08(O) 
single  input 

2  boards  hosted  on  a  backplane 


Serial  Interface 

Electrical 
Format 
Baud  rate 


RS232  or  RS422 

asynchronous 

9600 
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